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ABSTRACT: Design documents, drawings, and specifications are visual representations that are fundamental and 
prevalent in today’s construction engineering practice. Construction specialties (e.g., structural, mechanical) rely 
on these visual representations to express and draw meaning during collaborations. Construction engineering and 
management (CEM) students must acquire the knowledge, skills, and abilities — a key example of which is 
perceptual competence —for interpreting visual representations to facilitate efficient task execution, such as 
planning. Empowering learners with new technology using robust real-world immersion and interactive features 
is a significant step towards this target. The presented research explores new human-machine interactions to 
determine the best way for CEM students to learn through the combined senses of sight and touch. The approach 
merges visual and haptic interactions within an immersive environment to enhance perception and reasoning skills. 
The research demonstrates how CEM learners interact with and interpret the meanings of information within a 
planning task. It explores how VR and haptic technology augment the ability to recognize meanings — a new type 
of representational competency — for improved interpretation of information related to components with respect 
to engineering disciplines and sub-systems in a CEM, and investigates learners’ problem-solving ability by using 
perception-rich enhanced virtual reality (VR) and haptic affordances. 
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1. INTRODUCTION 


To satisfy the educational needs of STEM learners and foster essential 21st-century skills, such as critical thinking, 
reasoning, problem-solving, collaboration, and communication, educators must integrate innovative technology 
into the learning process (NSF, 2020). To address these requirements, human-computer interaction (HCI) offers 
viable solutions to augment human senses and enrich sensory input, including vision, hearing, smell, and touch 
(Manchanda et al., 2017). 


The sense of touch or haptics is one of the most informative human senses. This sense includes both cutaneous 
and kinesthetic sensations. Embracing haptics opens up new possibilities to expand human capabilities, such as 
improving manual dexterity and enhancing sensory perception (Chryssa & Julie-Ann, 2020). This research takes 
advantage of the HCI affordances and explores the use of haptic technology in learning for Construction 
Engineering and Management (CEM) students. 


Fundamentally, to explore the use of haptics in CEM learning, the presented approach draws on an individual’s 
spatial-temporal cognitive ability (STCA) (Mutis, 2018a). Spatial-temporal ability allows learners to effectively 
manage and comprehend significant amounts of spatial (how design components are related to one another in the 
3D space) and temporal (the logic in a process, such as the order, sequences, and hierarchies of the resources within 
a construction task) information (Mutis, 2018a). Limited or no ability to process spatial and temporal information 
(1.e., lack of spatial and temporal cognitive ability hinders the understanding of designs and management of the 
varying local conditions (e.g., unplanned conditions) (P. Antonenko & I. Mutis, 2017; P. D. Antonenko & I. Mutis, 
2017; Mutis, 2014, 2015; Mutis, 2018b). The ability helps learners to conceptualize three-dimensional 
relationships between objects in space and mentally manipulate them as sequential transformations over time. 


The STCA cognitive ability allows the CEM learners to recognize meanings and facilitates coupling observed 
representation to the given contexts — a new representational competency. The coupling abilities (spatial and 
temporal) significantly benefit the decision-making process. Individual spatial-temporal abilities are associated 
with high cognitive reasoning that defines the cognitive-processing chain — from basic visual attention to higher- 
level reasoning, such as an interaction between organizing, performing, and supervising the effectiveness of a plan 
(Mutis, 2018a). For instance, planning is a highly cognitively demanding task where STCA plays a pivotal role. 
Planning is critical as the learner couples observed representation in a given context to organize, perform, and 
supervise the effectiveness of a plan while interpreting information from engineering designs. Effective STCA 
training enables individuals to instantly identify concepts, events, and patterns for comprehension and projection, 
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streamlining actions, solutions, and implementations in planning. 


The presented approach explores the uses of haptic technology to augment cognitive capabilities, in particular the 
STCA. The STCA augmentation effect is from the cognitive load reduction by using a new sensing channel 
(haptics) in the cognitive process by liberating mental resources for other cognitive tasks (Sweller, 1988; Sweller 
et al., 2019), potentially enhancing spatial and temporal processes that are fundamental in problem-solving tasks. 
The assumption is that learners can rely on their haptic sense to reduce efforts of converting cognitive processes 
into physical actions— alleviating the burden of effort for processing spatial (e.g., spatial configurations of design 
components in the 3D space) and temporal (e.g., the logical sequence of design components for their assembly) 
information. The use of new senses (haptics) is a form of increasing the impact of embodied intervention in the 
cognitive process by, for instance, facilitating tracking information and gaining object rotations to feel and 
comprehend spatial relations more accurately (Tran et al., 2017). 


By using perception-rich enhanced virtual reality (VR) with haptic affordances, this study addresses the following 
questions: 

1. What aspects of haptic stimulus impact the learners’ development of representational competence for 
better interpretation of information related to designs in a CEM? The research outlines the importance of 
improving spatial-temporal skills to facilitate high-level reasoning in complex situations. 

2. What new HCI factors, combining visual and haptic (VH) interactions with engineering designs, enrich 
the perception and reasoning skills of CEM learners, leading to more accurate and efficient task 
execution? The solution presents a haptic language that implies tactile cues enhancing spatial awareness 
for the given context. 


2. BACKGROUND 


Researchers in STEM education are exploring the ways in which haptic technology can enhance the learning 
process, including improving student engagement, conceptual understanding, and skill acquisition. Early studies 
focused on developing haptic devices for enhancing spatial awareness and visualization skills (Liu et al., 2003; 
Williams Ii et al., 2001). Later research underlined the benefits of haptic feedback in improving interactions and 
spatial guidance (Jong, 2014; Takahashi et al., 2009). As demonstrated in further publications, augmenting VR 
with haptics increases overall task performance and the users’ perceived sense of presence (Cooper et al., 2018; 
Kreimeier et al., 2019). 


Over the years, haptic interventions in architecture, engineering, and construction (AEC) have been applied to 
simulate assembly tasks (Medellin-Castillo et al., 2015) and develop vocational training for construction personnel 
such as carpenters, plumbers, and masons (Jose et al., 2016; Ranjith et al., 2014). Current research aims to cultivate 
more sophisticated haptic devices and techniques for human-machine interaction in AEC, including haptic 
feedback for mixed reality and teleoperation (Adami et al., 2022). 


In general, haptics is extensively used in engineering learning, including training, physics and chemistry 
simulations, robotics, and automation (Prabhakaran et al., 2022; Sanfilippo et al., 2022). Engineering education 
utilizes haptic interfaces to provide students with hands-on experience with virtual simulations. Likewise, 
vocational training with haptics provides realistic practice in handling heavy machinery and tools. Lastly, by using 
haptic devices on remote-controlled construction robots, operators are able to discern the properties of various 
objects and materials during the manipulation (Alakhawand et al., 2022). Thus, haptic technology shows promise 
to transform traditional learning and training methods, offering advantages such as enhanced knowledge retention, 
engagement, skill acquisition, safety, and accessibility (Mastrolembo Ventura et al., 2022). 


Several studies have been conducted on assembly techniques, but only a few have explored the incorporation of 
haptics due to their relative novelty as an assistive tool in STEM learning. However, the development of haptics 
shows potential for enabling innovative approaches to enhance cognitive and motor skills, particularly in tasks like 
modeling, assembling, and teleoperation. For virtual assemblies, Yuan et al. (2008) introduced an augmented 
reality (AR) approach, utilizing a virtual interactive tool called VirIP and a visual assembly tree structure (VATS). 
This system enables assembly operators to seamlessly follow a pre-defined assembly plan/sequence without 
requiring sensor schemes or markers on the assembly components. Hu and Zhang (2012) presented a method 
leveraging a 3D game engine and software component technique to rapidly construct a reusable component library 
to develop virtual assembly experiments. In recent work, Li et al. (2020) proposed a framework with advanced 
computations such as runtime degrees of freedom (DOF) determination, disassembly directionality computation, 
and assembly/disassembly sequence generation. These computations efficiently integrate assembly constraint 
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information into a virtual assembly application with minimal effort required. 


Haptic technology allows the transfer of touch-based information between humans and computer interfaces (OED, 
2020). Haptics can enhance the learning experience and support an environment that cultivates student engagement, 
motivation, and interest in the subject matter (Tytler, 2020). Haptic interaction is crucial for a sense of presence 
and manipulating objects in remote or virtual environments with manual dexterity (Kortum, 2008, p. 25). For 
example, by providing users with tactile cues, haptics makes the digital environment more interactive and 
informative. 


In the AEC discipline, there are proposed haptic interventions that aim to assist users in accomplishing an 
engineering task providing guidance for the decision-making process. Rahimian and Ibrahim (2011) proposed a 
haptic-based VR 3D sketching interface to improve novice designers’ engagement with “problem-space” and 
“solution-space”, leading to increased artifact maturity in collaborative conceptual architectural design. Following 
Christiand and Yoon (2011) work, haptic-path sequence guidance reduces the assembly time and the travel distance 
that enhances the working performance of virtual assembly tasks. Also, the availability of haptics in large 
immersive environments can contribute to future advances in virtual assembly planning and factory simulation 
(Pavlik et al., 2013). Yeh et al. (2013) suggested that multi-symbolic representations (text, digits, and colors) in 
haptics-enhanced virtual reality systems have the potential to help collaborative work effectively. James et al. 
(2019) proposed a bi-manual haptic interface for skill acquisition in surface mount device soldering. Coffey and 
Pierson (2022) demonstrated the effectiveness of the proposed haptic guidance system for co-navigation of non- 
holonomic vehicles through teleoperation. Williams et al. (2023) presented a framework for active haptic guidance 
in mixed reality using one or more robotic haptic proxies to influence user behavior and deliver a safer and more 
immersive virtual experience. 


The primary focus of the mentioned studies was to improve the understanding of a process for training (e.g., the 
process of assembling building components). The studies have incorporated haptic guidance into the assembly 
processes, which helped users receive tactile feedback during the assembly tasks. However, the haptic guidance 
implementations fell short in providing spatial awareness and addressing high-order cognition in cognitively 
demanding tasks such as identification of the dependencies or hierarchy of building components for planning. 
While the haptic guidance aids in recognizing information about movements in training tasks through haptic 
feedback, the approaches do not offer a comprehensive understanding of the entire spatial context or 
interconnections between various building components. The presented study aims to overcome these limitations 
by exploring spatial-temporal cognitive abilities using visual and haptic stimuli. 


Haptic feedback 


Using electronic devices, we encounter multiple interactions, including sounds, flashes, and buzzing haptics 
(Müller, 2020). Such a combination of sensory stimuli allows the user to be fully engaged in the experience, which 
enriches the overall quality of the interaction. A crucial aspect of this set is haptic feedback, which draws from the 
psychological nature of interaction with the environment and other humans (e.g., social touch). Therefore, 
achieving precise replication of haptic signals in devices requires a deep comprehension of how humans perceive 
and attribute meaning to tactile interactions to portray their semantics accurately. 


The human skin’s discriminative ability arises from a dense network of cutaneous receptors allowing us to 
differentiate fine touch, pressure, texture, and temperature (Fulkerson, 2020). This adaptability of touch perception, 
known as adaptation rate, enables us to prioritize novel sensations while filtering out constant stimuli. Unlike some 
other senses perceived passively, haptic perception is inherently interactive and bidirectional — we actively explore 
and manipulate the environment to extract tactile details. 


To recreate physical sensations, HCI incorporates various types of haptic technology, including force, vibrotactile, 
ultrasonic, thermal, and other forms of haptic feedback (Hatzfeld et al., 2015). Haptic interfaces allow users to 
experience tactile sensations while manipulating objects, discriminating textures, and applying forces in the virtual 
and physical environment. 


According to the literature (Adilkhanov et al., 2022), haptics performs three primary functions such as simulation, 
teleoperation, and guidance. Through simulations, haptic feedback imitates physical interaction with the 
environment and its attributes to heighten the realism of learning scenarios. In teleoperation, the haptic interface 
provides a two-way communication channel between a robot and an operator, allowing the operator to perceive 
tactile feedback from the robotic tool (Luo et al., 2019). As part of the guidance process, haptics implement tactile 
patterns to derive directional cues to the user (Huang et al., 2019). 
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Guiding haptics becomes especially beneficial for facilitating the decision-making process and fostering problem- 
solving abilities by providing tactile cues to assist users in performing tasks or enhancing interactions in a physical 
and virtual environment (Bluteau et al., 2008; Feygin et al., 2002). This haptic function utilizes touch-based 
sensations to provide the users with real-time information, helping them make informed decisions and improving 
their overall performance and understanding of the context. Research suggests that using intuitive haptic guidance 
to assist the movement reduces errors (Mugge et al., 2016). Moreover, a partial-then-full haptic guidance strategy 
seems the most effective in improving learning outcomes (Teranishi et al., 2018). The most common applications 
of guiding haptics include vibrotactile feedback, often incorporated into commercial smartwatches for haptic 
notifications and alerts. 


Haptic guidance can be achieved through a haptic code that utilizes touch-based symbols (e.g., haptic icons or 
“hapticons” (Enriquez & MacLean, 2003)) to instantly deliver information to the user via vibrations, pressure, or 
movement (Hatzfeld et al., 2015, p. 75). According to Enriquez et al. (2006), haptic code has to meet the following 
conditions in order to offer explicit meaning: 
e Differentiable: All haptics must be distinct from one another when presented either alone or in any 
common haptic combinations. 
e Identifiable: Once a meaning has been connected to a stimulus to form an icon, it must be simple to recall. 
e Learnable: The associations between meanings and stimuli should be intuitive and easily remembered. 
The elementary functions of the haptic code include providing notifications with neutral feedback and 
signals with either positive or negative meaning in response to the user’s actions. 


Haptic code can be applied even on a broader spectrum, e.g., for rendering abstract models or concepts as a new 
modality for communication. At the lowest level, haptic devices notify users of an event, their identity, or their 
current state or contents. A higher level of abstraction implies haptic associations that allow the users to identify 
interdependencies and determine a sequence of actions by assigning physical sensations to an object hierarchy. 
Accordingly, systematic, perceptually guided haptic design can support expressive and nuanced communication 
that qualifies as a new haptic language. 


3. METHODOLOGY AND APPROACH 


The study consists of two main phases: (1) the creation of the experimental training platform, designed to be 
interactive and informative; (2) experimentation, with active student participation for practical application and 
assessment of the learning outcomes. This comprehensive approach provides an effective and engaging program 
for students to develop their skills and comprehend complex building concepts in a virtual and immersive 
environment. The presented research is the first phase, including a case example to illustrate the approach. 


Immersive virtual platform 


The VR design consists of the development of a VR environment based on the detailed design of a building project 
(e.g., a small residential building). See Fig 1. The design was represented in a Building Information Model (BIM) 
with at least a Level of Detail 300. The BIM model contains rich data on engineering systems through represented 
objects or component assemblies, such as quantity, size, shape, location, and orientation. The design was exported 
as an Industry Foundation Class (IFC) file to preserve the semantic information of the building components. The 
exported model was then imported into Unity for two purposes. First, it acts as a reference point in the form of a 
translucent building, allowing the user to place building components accurately. Second, it is semantically broken 
down into corresponding building components to build game objects. The resulting structures were game objects 
created based on the standard categorization of the building into Sub-Structure and Superstructure and further 
classified into Structural, Architectural, and Mechanical components. 


It is critical to note that the created game objects were set for true building scale, generating an immersion that 
represents dimensions for easy manipulation in the VR environment. Each game object had a representation that 
described data and text information in a structured format, involving attributes as game elements based on IFC 
structure. For example, each object game had data related to the activity (used for planning) in their element 
attributes (element descriptors). The element attributes held in addition to the planning activity information 
associated with unique haptic feedback, as discussed in the section below. For a logical representation in planning, 
game objects were nested based on a work breakdown structure (WBS)— a hierarchical tree structure subdividing 
the deliverables and work. The WBS disciplines will deliver the work specified in each work package—the lowest 
level in the WBS that represents a specific amount of work. The work package as product and deliverable has a 
VR object representation. The structure of these components is shown in Fig. 1. 
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Fig. 1: Design structure of the VR platform. 


Experimental Platform 


Unity software integrated with a VR headset Oculus Pro and a full set of haptics devices (see 2) were used for the 
development. This state-of-the-art platform provides users with a fully immersive experience. An example of the 
visualizations is shown in Fig. 5a and Fig. 5b. They illustrate a dashboard and virtual design components where 
users learn virtual manipulation, featuring an informative activity pane to hold building components as activity 
tiles and servicing as a comprehensive reference model for planning activities to enhance the overall learning 
experience. 


Fig. 2: Haptic devices for the VR platform. 
Haptic (vibrotactile) code 


The researchers systematically structured the haptic code as feedback for the simulation and experimentation in 
VR environment. The code contains the logical patterns that guide the user’s manipulation of the building 
components through interaction augmented by haptic feedback. The code has signatures expressed as haptic icons, 
i.e., a haptic icon is a brief haptic stimulus associated with meanings. The haptic icons were designed to intuitively 
comprehend cues about a function of the object and interact (user-object effects) in the virtual environment. The 
code is a form of primary language wherein each icon is a constant pattern with associated semantics. The learners 
(users) are required to get familiarized with the code (akin to learning a primary language to operate a system) a- 


priory. 
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TION A - EXTE ALITY TECHNOLOGIES IN CONS" 


To associate semantics to the haptic code, four key perceiving haptic features play a crucial role in defining the 
tactile experience: 

e Intensity. It governs the strength or magnitude of the tactile sensation delivered to the user. It determines 
how strong or weak the haptic feedback feels, allowing for the creation of subtle or intense tactile 
perceptions. 

e Sharpness. It relates to the perceived abruptness or distinctness of the haptic sensation. It influences 
whether the sensation feels smooth or sudden. 


e Duration. It refers to the length of time of the perception of haptic feedback. Short durations can convey 
quick events, while longer durations can simulate prolonged interactions or sustained sensations. 


e  Granularity. It is determined by the frequency of impulses and their spacing. The more granularity, the 
more rapid the impulses. 


Manipulations with these haptic features enable prototyping and fine-tuning haptic experiences to match specific 
interactions and simulation scenarios, enhancing user engagement and immersion in virtual environments. The 
combinations of the haptic features assigned to a haptic device evolve into distinctive haptic patterns. 


Haptic code (vibrotactile) types 


The haptic code consists of two types of haptic feedback: operational and functional. 
Operational 


It refers to haptic feedback of the basic human-computer interaction (HCI) with the elements of the virtual 
environment, such as feedback on actions on the system components (to select, cancel, move, etc.). The approach 
includes three types of operational haptic feedback: 
e Positive to reflect the correct actions of the user by giving soft impulses with low or medium intensity; 
e Negative to associate the mistakes and has more even rigid impulses, medium or high intensity; 
e Neutral to provide alerts to the user regarding updates or notifications (it is presented as a row of short 
impulses with gaps in between). 


Functional 
It refers to the feedback that gives semantics associated with activity planned in VR deployment. 


Parameters of duration (D), granularity (G), intensity (I), and sharpness (S) define the functional haptic code. The 
combination of parameters defines features that indicate semantics. The combination can be represented in a two- 
dimensional matrix of n rows (where n is the number of combinations). See Fig. 3. Each row represents the 
distribution of values of parameters (D, G, I, S). 


A VR object will have an associated haptic code combination (DGIS), representing a specific value and semantics. 


Fig. 1 illustrates the approach conceptualization of the intersecting components (virtual environment, structure of 
VR objects, haptic (vibrotactile) code, semantics haptic feedback (as semantics), and the spatial temporal cognitive 
ability (while interacting with problem solving in CEM). The arrangement impacts the spatial-temporal cognitive 
abilities of learners, assisting them in accurately defining the sequence of activities through the integration of 
visual and tactile cues. 
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Fig. 1: Approach conceptualization. 


Case Example 


Learners are required to plan the construction of a small building design section in the VR environment (see Fig. 
2). The user should build the plan by identifying construction work packets (associated components and activities). 
A work package (construction product deliverable) serves to establish a coherent and feasible subdivision of tasks 
within the construction project. Each packet has associations with physical areas (work zones) to cover all the 


components of the design. 


A work breakdown structure (WBS) that incorporates the components and activities associated with the small 
building design (see Fig. 2a) is presented as a dashboard in the VR environment (see Fig. 5a). The WBS is used as 
a baseline for planning. The first milestone is set for substructure completion of the building design, and the second 
is set for the superstructure (see Fig. 2b). Each building component from the design is the deliverable ofan activity. 


The assembly sequence for each activity and packet (construction product deliverable) is based on the Finish-To- 
Start (FS) inference (logical relationship between two activities). A finish-to-start relationship implies that the 
predecessor activity needs to be finished before any subsequent actions can start. 
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Fig. 2: Construction product deliverables from work breakdown structure (WBS). 
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SECTION A - EXTENDED REALITY TECHNOLOGIES IN CONSTRUCTION 


After the user selects the packet from the dashboard in the virtual environment (see Fig. 5a), the next step is to 
select, drag, and drop the design component (deliverable) of the small building on a virtual layout by performing 
a virtual walkthrough (see Fig. 5b). 


(a) A dashboard servicing as a reference model to visualize (b) Snapshot of the mapping between the work package and the 
WBS packages for conceptual planning. reference model (virtual layout of the building). 


Fig. 5: Interactive haptic activity for a planning task in the VR environment. 


The order of activities corresponds to the deliverable sequence, and each deliverable has an associated location in 
the virtual layout. The assumption for the case example is that the presented activities depend on the completion 
of others before they can begin (FS precedence). Planning these activities takes the dependencies (precedents) into 
account by arranging activities in a logical sequence. The arrangement of all deliverables is the planning of the 
construction section of a small building design in the VR environment. 


The users need to locate (by dragging and dropping) all the deliverables of the building section in the virtual layout 
space during the virtual walkthrough. By completing all the packets in the dashboard, the user can complete the 
planning of the building. 


Haptic feedback is an interactive feature that responds to the actions of the users within the VR environment — 
i.e., certain actions generate a type of haptic feedback with associated code (meaning). When the user drags and 
drops a deliverable on its selected location, there are two potential haptic feedback: functional and operational. 
Thus, operational and functional haptics complement each other to assist with the understanding of the semantics 
and ensure proper placement of the system components. Operational haptic feedback is on basic human-technology 
interaction, while functional haptics are systematically organized and tailored to specific semantics that indicate 
hierarchical structures. 


For example, if the deliverable is placed correctly, operational haptic feedback (positive operational feedback using 
soft impulses with low or medium intensity) would indicate a code that will inform the user that the correct location 
was correctly selected. However, if it is misplaced, operational feedback with the associated code (negative 
operational feedback using rigid impulses, medium and high intensity) is given to indicate to the user the error of 
displacement. Another example of operational feedback is positive when the user reaches a designated milestone 
while finalizing the packets from the WBS. Otherwise, negative feedback is given — indicating that more 
selections are required for planning. Operational haptic (vibrotactile) is produced by the haptic sleeves, which 
offer feedback for component manipulations like selection and canceling, as well as the haptic vest and feet, which 
are responsible for delivering notifications, success signals, and failure alerts. Code examples of operational 
feedback are shown in Table 1. 
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Table 1: Operational haptic feedback code 


Events Meaning Haptic feedback Intensity Duration [ms] 

Select Medium intensity, medium sharpness, low 0.2 short 150 
A component is taken and short duration, two short impulses F 

medium 0.5 short 200 


with increasing intensity 


Cancel A component is thrown Medium intensity, medium sharpness, medium 0.4 short 100 


and short duration, two short impulses low 0.2 short 150 


with decreasing intensity 


Notification Generating another component Medium intensity, medium sharpness, medium 0.4 short 50 


two short impulses 


Error Implementation ofa component High intensity, sharp vibration, high 1.0 middle 400 
meets the constraints medium duration 

Success The component is applied Short burst of impulse, high intensity, high 0.7 short 100 
medium sharpness high 0.7 short 150 

Epic success A milestone is accomplished High intensity, medium sharpness high 0.7 Short 100 
Failure A task is failed Five short bursts of impulse with high 1.0 short 200, 
overlay, max intensity, high sharpness 250, 

300 


Functional haptic feedback provides semantics related to reasoning in problem-solving, involving analytical tasks 
for planning. Of particular interest is the user’s understanding of the relationships between design components in 
the physical space. An example of a relationship is the priority for construction, assembly, or installation of the 
design components in the physical space. Reasoning on the relationship demands spatial and temporal cognitive 
abilities (STCA). The aim of functional haptic feedback is to assist the user’s reasoning (spatial and temporal 
reasoning) when required. An example is providing a better comprehension or awareness of the order for 
construction and assembly among two or more design components—by featuring STCA— as shown in Fig. 5a 
and 5b. 
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Fig. 6: Spatial and temporal reasoning on design components (using STCA). 


Functional haptic feedback informs the user which elements possess the highest priority for their construction and 
assembly (i.e., some design elements have higher priority than others to make their construction feasible and 
efficient). ga, for instance, illustrates the building components— without any spatial and temporal information. 
Fig. b illustrates spatial and temporal information — the relationship among the objects in the physical space, by 
establishing the priority and order for their construction. Haptic code will help the learner to reason on spatial and 
temporal information using a combination of duration (D), granularity (G), intensity (I), and sharpness (S) features. 
For example, a combination of values from the parameters D, I, and S will inform the order distribution in a 
spectrum (e.g., from the lowest to the highest value or from the highest to the lowest value). Consequently, each 
component on the final level has its unique haptic code (DGIS) comprising values for each parameter. 


The functional haptic (vibrotactile) feedback is related to information on the hierarchy of construction activity 
sequencing. Interaction with each component is assigned with unique feedback, which allows the user to easily 
discriminate the components one from another based on their semantics by selecting them from the WBS of the 
building (Fig. 2b). Due to the perceptive haptic nature of hands, functional haptics is assigned to the haptic gloves. 
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4. CONCLUSION 


The presented study describes an exploration of new human-machine interactions to determine the effects of 
learning through the combined visual and haptic modalities in VR environments. The interactions with an 
immersive environment involve engineering design comprehension for planning activities— framed in a problem- 
solving task. The study presents the technology environment using VR and real-time haptic feedback for 
experiencing problem-solving tasks — by complementing semantics of visualizations (e.g., 3D designs) with 
haptic feedback (e.g., vibrations) for a CEM task. 


The approach to building a VR environment with dual interactive mode (visual and haptic) facilitates the creation 
of new forms of understanding problems in planning, a highly cognitively demanding task where STCA plays a 
pivotal role. Learners map VR visual and haptic features to domain (CEM) problems and build solutions to the 
planning problem. They used VR technology (headset and controllers) to engage embodied perceptuomotor 
information by interacting with visual and haptic representations. For example, users navigate the 3D design in 
VR to approach locations of interest, allowing iterations between representations and reflection while problem- 
solving. In future work with a higher number of testing subjects, it is expected to demonstrate that haptic feedback 
(haptic code) effectively informs the learners of the semantics of the components for the planning task, enabling 
the learner to infer conditions in a virtual scene. 


The technology’s pedagogical features will make design information from multiple engineering specialties readily 
available for haptic and visual perception in a stepwise process to learn planning tasks. The technology will 
facilitate learning through observation and VR movements of design components. The approach uses work packets 
(construction product deliverables) that would enable scenarios of learning about understanding deliverables as 
chunks of workload for planning—the smallest unit that can be planned and managed for construction operations. 
By enabling learning with a work packet focus, the approach facilitates understanding of planning by framing 
control into a process (set of steps for delivery) of construction (assembly). The method provides opportunities for 
the learner to assimilate complex simulated realities of the physical space and develop spatial-temporal cognitive 
ability. Spatial-temporal ability allows learners to effectively manage and comprehend significant amounts of 
spatial (how design components are related to one another in the 3D space) and temporal (the logic in a process, 
such as the order, sequences, and hierarchies of the resources within a construction task) information. 


The insights collected from this study underscore the significant potential of the VR and haptic cues to enhance 
the learners’ perception of a problem’s conditions that are not visible to the learner. Further exploration of 
technology experimentations will allow researchers to draw conclusions on the learners’ perceptual competence 
and problem-solving capabilities, thereby contributing to the formation of project engineers with high levels of 
productivity in the construction industry. 
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